Skip to content

Rewrite on welcome to lf section#2872

Merged
annabellscha merged 4 commits intolangfuse-academy-researchfrom
Rewrite-on-welcome-to-LF-section
Apr 29, 2026
Merged

Rewrite on welcome to lf section#2872
annabellscha merged 4 commits intolangfuse-academy-researchfrom
Rewrite-on-welcome-to-LF-section

Conversation

@annabellscha
Copy link
Copy Markdown
Contributor

@annabellscha annabellscha commented Apr 27, 2026

Disclaimer: Experimental PR review

Greptile Summary

This PR rewrites the Langfuse Academy welcome section: index.mdx is streamlined with clearer framing and the old TODO callouts are removed, a new ai-engineering-loop.mdx page is added as the conceptual entry point for the lifecycle, and three meta.json files are updated to use gerund-form titles ("Building Datasets", "Experimenting", "Evaluating") for consistency.

Confidence Score: 5/5

Safe to merge — documentation-only changes with no broken links or rendering issues.

All changed files are MDX content and JSON navigation config. All internal links in the new page resolve to existing Academy sections. The only findings are minor whitespace style issues (P2). No P0 or P1 issues found.

No files require special attention.

Important Files Changed

Filename Overview
content/academy/ai-engineering-loop.mdx New page introducing the AI Engineering Loop concept; all 5 card links resolve to existing Academy sections; the TODO comment is a JSX block comment and will not render in production.
content/academy/index.mdx Welcome page rewritten with cleaner framing; previous TODO callouts removed; minor whitespace inconsistencies (extra blank lines, missing blank line before last paragraph).
content/academy/meta.json New ai-engineering-loop page correctly registered in the navigation before the ---The Loop--- separator.
content/academy/datasets/meta.json Section title updated from Datasets to Building Datasets to align with the gerund naming convention used across the loop.
content/academy/evaluate/meta.json Section title updated from Evaluate to Evaluating for consistency.
content/academy/experiments/meta.json Section title updated from Experiments to Experimenting for consistency.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([Academy Index\nindex.mdx]) --> B([AI Engineering Loop\nai-engineering-loop.mdx])
    B --> C([Tracing\n/academy/tracing])
    B --> D([Monitoring\n/academy/monitoring])
    B --> E([Building Datasets\n/academy/datasets])
    B --> F([Experimenting\n/academy/experiments])
    B --> G([Evaluating\n/academy/evaluate])
    A --> C
    A --> D
    A --> E
    A --> F
    A --> G
Loading

Comments Outside Diff (2)

  1. content/academy/monitoring/overview.mdx, line 598-602 (link)

    P1 Internal author notes visible in production

    This <Callout> block contains raw internal working notes that will render as visible content for site visitors. The IMPORTANT: annotation and the "Add more details…" instruction are author reminders, not reader-facing content. The same pattern appears across several other Academy pages (e.g. datasets/overview.mdx has three TODO callouts, evaluate/overview.mdx has three, tracing/overview.mdx has three, experiments/overview.mdx has two). All of these will be publicly visible as rendered callout components until they are resolved or removed.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: content/academy/monitoring/overview.mdx
    Line: 598-602
    
    Comment:
    **Internal author notes visible in production**
    
    This `<Callout>` block contains raw internal working notes that will render as visible content for site visitors. The `IMPORTANT:` annotation and the "Add more details…" instruction are author reminders, not reader-facing content. The same pattern appears across several other Academy pages (e.g. `datasets/overview.mdx` has three TODO callouts, `evaluate/overview.mdx` has three, `tracing/overview.mdx` has three, `experiments/overview.mdx` has two). All of these will be publicly visible as rendered callout components until they are resolved or removed.
    
    How can I resolve this? If you propose a fix, please make it concise.
  2. content/academy/monitoring/overview.mdx, line 608 (link)

    P1 Broken link to non-existent page

    /academy/error-analysis is linked here (and referenced inline at line 576 as "covered in depth in error analysis"), but no content/academy/error-analysis/ directory or MDX file exists in this PR and the topic does not appear in content/academy/meta.json. Clicking this link will produce a 404 via the AcademyNotFound component.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: content/academy/monitoring/overview.mdx
    Line: 608
    
    Comment:
    **Broken link to non-existent page**
    
    `/academy/error-analysis` is linked here (and referenced inline at line 576 as "covered in depth in error analysis"), but no `content/academy/error-analysis/` directory or MDX file exists in this PR and the topic does not appear in `content/academy/meta.json`. Clicking this link will produce a 404 via the `AcademyNotFound` component.
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: content/academy/index.mdx
Line: 14-16

Comment:
**Extra blank lines before first section heading**

There are three consecutive blank lines between the introductory paragraph and `## What you will find here`. This won't break rendering, but it's inconsistent with the rest of the file and the other Academy pages.

```suggestion
Langfuse Academy maps the AI engineering lifecycle so you understand how the pieces fit and what it takes to ship from prototype to production.

## What you will find here
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: content/academy/index.mdx
Line: 41-42

Comment:
**Missing blank line between heading and paragraph**

`## Why we are publishing this` has no blank line separating it from its paragraph text. Most Markdown parsers handle this gracefully, but adding a blank line is conventional and avoids any parser edge cases.

```suggestion
## Why we are publishing this

Langfuse is open source, and we want to open source the conceptual side of AI engineering too. The Academy is our way of making the core ideas, vocabulary, and workflows behind LLM application development easier to access for everyone.
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (2): Last reviewed commit: "Merge branch 'langfuse-academy-research'..." | Re-trigger Greptile

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 27, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
langfuse-docs Ready Ready Preview, Comment Apr 29, 2026 0:41am

Request Review

@dosubot dosubot Bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Apr 27, 2026
@github-actions
Copy link
Copy Markdown

@claude review

@dosubot dosubot Bot added the documentation Improvements or additions to documentation label Apr 27, 2026
@annabellscha annabellscha reopened this Apr 27, 2026
@annabellscha annabellscha marked this pull request as draft April 27, 2026 09:02
@annabellscha annabellscha changed the base branch from main to langfuse-academy-research April 27, 2026 09:03

Across the loop, teams are usually balancing three things at once: output quality, latency, and cost. The point is not to optimize one number in isolation, but to make tradeoffs explicit and grounded in evidence from your own application.

## Where the docs fit
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove here

Comment thread content/academy/index.mdx Outdated
That changes what teams need to understand and manage. It is no longer enough to ask whether the system ran. You also need a way to reason about output quality, iteration, and the tradeoffs that come with shipping AI products.

## Why LLM observability is different
Langfuse Academy exists to help you build that mental model. It maps the AI engineering lifecycle step by step so you can understand how the pieces fit together and what it takes to move from prototype to production.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swap mental model with understanding.

Comment thread content/academy/index.mdx Outdated
Building with LLMs changes the job of engineering teams. Once outputs become probabilistic, a system can be technically healthy and still produce responses that are wrong, incomplete, off-brand, unsafe, or simply not useful.

Rather than focusing on individual product features, Academy is meant to help you understand the bigger picture, and how teams can work with that change in a systematic way.
That changes what teams need to understand and manage. It is no longer enough to ask whether the system ran. You also need a way to reason about output quality, iteration, and the tradeoffs that come with shipping AI products.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change this to output, quality, cost, latency, and their trade-offs that come with AI shifting. With shipping AI products


The loop is a working model, not a strict waterfall. Teams move through it repeatedly, and different parts of the loop become more important as a product matures.

## The steps
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, let's try to make all steps actionable, so:

  • Tracing
  • monitoring
  • building datasets
  • experimenting
  • evaluate
    Something like that, so it's all the kind of same word.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure to propagate

Comment thread content/academy/index.mdx Outdated
## Why we are publishing this

But LLM applications introduce a different kind of challenge. Their behavior is probabilistic: the same input can produce different outputs, and a response can look plausible even when it is wrong, incomplete, off-brand, unsafe, or simply unhelpful. In other words, a request can succeed technically and still fail for the user.
Langfuse is open source, and we want to open source the conceptual side of AI engineering too. The Academy is our attempt to make the mental models, vocabulary, and workflows behind LLM application development easier to access for everyone.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is our way of making mental models, vocabulary, and workflows behind LLM application development easier to access for everyone

Comment thread content/academy/index.mdx Outdated
- Software engineers moving into AI product development
- Product managers who need to reason about quality, iteration, and tradeoffs
- People learning the field and trying to understand the core concepts
- Technical and business leaders who need a working model of how AI systems are built and improved
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agents that help the humans figure out LLM engineering.

Comment thread content/academy/meta.json
"title": "Academy",
"pages": [
"index",
"ai-engineering-loop",
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's here, but it's weird that the AI engineering loop is nested in the AI engineering loop. It should just be one tab that I click on, and then I get the content; nothing to unfold.


## The steps

### 1. Tracing
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I like the content, it's visually difficult because there is so much text. Can we somehow either bullet point or shorten it without losing a lot of content, because I think it gives the high-level overview so I

Copy link
Copy Markdown
Contributor Author

@annabellscha annabellscha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please implement all commands.

Comment on lines +18 to +34
Tracing captures the full path of a request so you can inspect prompts, retrieved context, tool calls, outputs, latency, and cost in one place. Read [Tracing](/academy/tracing) for a breakdown of what a useful trace looks like and why traces become the foundation for everything else.

### 2. Monitoring

Monitoring turns raw traces into ongoing visibility by tracking trends and surfacing the cases that deserve attention. Read [Monitoring](/academy/monitoring) to understand how teams watch quality, cost, latency, and production failures over time.

### 3. Datasets

Datasets turn real scenarios into repeatable test cases so you can check whether a change helps across more than a handful of examples. Read [Datasets](/academy/datasets) for how to structure dataset items and when it makes sense to split or grow a dataset.

### 4. Experiments

Experiments let you change one variable at a time and compare outputs against a stable baseline instead of relying on intuition alone. Read [Experiments](/academy/experiments) to see how to isolate variables, compare variants, and learn what actually improved.

### 5. Evaluation

Evaluation is how you decide whether results are good enough to ship, using manual review, code-based checks, or LLM judges depending on the task. Read [Evaluate](/academy/evaluate) for how teams score outputs and turn qualitative judgments into a repeatable process.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 All primary CTA links in the new ai-engineering-loop/overview.mdx and the rewritten index.mdx will 404. Each academy subsection contains only overview.mdx (no index.mdx), so under fumadocs these resolve at /academy/<section>/overview, not /academy/<section> — and unlike the docs section, no equivalent redirects exist in lib/redirects.js. Fix by either appending /overview to each link, or adding /academy/* entries to lib/redirects.js:60-71 analogous to the existing docs redirects.

Extended reasoning...

What's broken

Every primary call-to-action link added by this PR points to a bare /academy/<section> URL, but those URLs do not resolve. The affected links are:

content/academy/index.mdx:

  • Line 30: [The AI Engineering Loop](/academy/ai-engineering-loop)
  • Lines 32–36: /academy/tracing, /academy/monitoring, /academy/datasets, /academy/experiments, /academy/evaluate

content/academy/ai-engineering-loop/overview.mdx:

  • Line 18: [Tracing](/academy/tracing)
  • Line 22: [Monitoring](/academy/monitoring)
  • Line 26: [Datasets](/academy/datasets)
  • Line 30: [Experiments](/academy/experiments)
  • Line 34: [Evaluate](/academy/evaluate)

Why they 404

I verified directly that each academy subfolder (ai-engineering-loop, tracing, monitoring, datasets, experiments, evaluate) contains only meta.json + overview.mdx — no index.mdx. Under fumadocs-core's default source loader, a file at content/academy/tracing/overview.mdx is exposed at URL /academy/tracing/overview. There is no automatic 'use overview as folder index' fallback — only an index.mdx in a folder takes the bare folder URL. So academySource.getPage(["tracing"]) returns undefined, and app/academy/[[...slug]]/page.tsx:14 falls through to notFound(), rendering AcademyNotFound.

The repo's own precedent confirms this

lib/redirects.js:60-71 explicitly maps these docs paths to their /overview children:

...[
  "/docs/integrations",
  "/docs/scores",
  "/docs/datasets",
  "/docs/security",
  "/docs/observability",
  "/docs/evaluation",
  "/docs/metrics",
  "/docs/api-and-data-platform",
  "/docs/prompt-management",
].map((path) => [path, path + "/overview"]),

These redirects exist precisely because those docs sections have the same structure (only overview.mdx, no index.mdx) and the bare folder URLs do not resolve on their own. Grepping lib/redirects.js for 'academy' returns only an unrelated /customers/khan-academy entry — no /academy/* redirect was added by this PR.

Step-by-step proof

  1. User lands on /academy (works — content/academy/index.mdx exists at the root, mapped to /academy).
  2. User clicks the first CTA, [Tracing](/academy/tracing), on line 32 of the new index.mdx or line 18 of ai-engineering-loop/overview.mdx.
  3. Next.js routes the request to app/academy/[[...slug]]/page.tsx with slug = ["tracing"].
  4. academySource.getPage(["tracing"]) looks for a page whose URL path segments are ["tracing"]. The only file under content/academy/tracing/ is overview.mdx, which fumadocs maps to slug ["tracing", "overview"] — no match.
  5. Result is undefined, so if (!page) notFound(); triggers.
  6. The user sees the AcademyNotFound 404 page.

The same chain applies to all six bare-folder URLs introduced by this PR. The PR landing page itself loads, but the moment a reader clicks any 'next step' link, the journey breaks.

Impact

Every primary CTA in the two pages this PR adds/rewrites is broken. This is a release blocker for the Academy section: the rewritten welcome page exists specifically to send readers to the loop sub-sections, and the new ai-engineering-loop/overview.mdx is structured around five 'Read [section]' links that all 404. Greptile flagged a similar broken link to /academy/error-analysis in the pre-existing monitoring page but missed this systemic issue across the new content.

Fix

Two equivalent options:

  1. Append /overview at each link site — change /academy/tracing/academy/tracing/overview, etc., in both files (matches what lib/nav-links.tsx already does for the docs section).
  2. Add redirects to lib/redirects.js analogous to lines 60–71:
...[
  "/academy/ai-engineering-loop",
  "/academy/tracing",
  "/academy/monitoring",
  "/academy/datasets",
  "/academy/experiments",
  "/academy/evaluate",
].map((path) => [path, path + "/overview"]),

Option 2 is more robust because it also catches any future external links that target the bare folder URL.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially addressed. Moving ai-engineering-loop to a top-level .mdx did fix the /academy/ai-engineering-loop link, but the other five CTAs still 404:

  • content/academy/index.mdx:20-24: /academy/tracing, /academy/monitoring, /academy/datasets, /academy/experiments, /academy/evaluate
  • content/academy/ai-engineering-loop.mdx:25,33,48,56,64,86: same five plus the Start with Tracing CTA

Each of these subfolders still only contains overview.mdx, and no /academy/* entries were added to lib/redirects.js (only the unrelated /customers/khan-academy mapping exists). Either append /overview at every link site, or add the same kind of redirect block as lib/redirects.js:60-71:

...[
  "/academy/tracing",
  "/academy/monitoring",
  "/academy/datasets",
  "/academy/experiments",
  "/academy/evaluate",
].map((path) => [path, path + "/overview"]),

@annabellscha annabellscha marked this pull request as ready for review April 29, 2026 00:37
@annabellscha annabellscha merged commit 66bf0c9 into langfuse-academy-research Apr 29, 2026
8 of 10 checks passed
@annabellscha annabellscha deleted the Rewrite-on-welcome-to-LF-section branch April 29, 2026 00:37
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Apr 29, 2026

<Cards num={3} className="gap-6">
<Cards.Card
title="3. Building datasets"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 On line 47, the card title reads title="3. Building datasets" (lowercase d), but the corresponding sidebar label in content/academy/datasets/meta.json and the link in content/academy/index.mdx:22 both read Building Datasets (uppercase D). A reader sees "Building datasets" on the card and then "Building Datasets" on the destination page. Suggest changing this to title="3. Building Datasets" to match.

Extended reasoning...

What the bug is

The new AI Engineering Loop page introduces five cards linking to the loop sub-sections. Four of those cards have single-word titles (Tracing, Monitoring, Experimenting, Evaluating) where casing is unambiguous. The one multi-word card — the third one, for datasets — uses sentence case on the card but Title Case everywhere else in the same PR.

The specific code path

  • content/academy/ai-engineering-loop.mdx:47title="3. Building datasets" (lowercase d).
  • content/academy/datasets/meta.json"title": "Building Datasets" (uppercase D). This drives the sidebar entry and the page heading.
  • content/academy/index.mdx:22[Building Datasets](/academy/datasets) (uppercase D). This is the link in the welcome page bullet list.

So the card on the loop page is the only place that uses lowercase d, and it diverges from both the sidebar label of the page it links to and the sibling link in the welcome page list.

Why existing code does not prevent it

There is no shared constant or central source of truth for these section titles — each is hand-typed. Nothing in the build will warn when a card label diverges from its target page title. The other four cards happen to be safe because their titles are single words.

Step-by-step proof

  1. Reader opens /academy/ai-engineering-loop and sees five cards. The third reads "3. Building datasets".
  2. Reader clicks the card, which navigates to /academy/datasets.
  3. fumadocs renders the page using content/academy/datasets/meta.json, whose title is "Building Datasets".
  4. The destination page header and sidebar entry both display "Building Datasets" — different casing from the card the reader just clicked.
  5. Separately, on /academy (the index page), the bullet list at line 22 shows Building Datasets, also disagreeing with the loop card.

Impact

Purely cosmetic — no broken navigation, no functional issue. But it is the only multi-word card and the only one whose label disagrees with its destination, so the inconsistency is asymmetric and stands out.

Fix

Change content/academy/ai-engineering-loop.mdx:47 from title="3. Building datasets" to title="3. Building Datasets" to align with datasets/meta.json and index.mdx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant